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This  report  contains  additional  material  that  is  not  contained  in  the  Phase  II 
application. 


Phase  I  Objectives 


The  research  during  Phase  I  had  three  major  objectives. 

The  first  aim  was  the  assembly  of  a  small  prototype  neural  computer  and  its  interface 
with  a  digital  host  computer.  It  should  be  emphasized  that  because  of  its  limited  size  the 
Phase  I  prototype  was  not  designed  for  practical  applications.  Instead  it  was  intended  as 
a  platform  for  checking  the  basic  design  concept  as  well  as  the  function  of  the  individual 
components. 

The  second  objective  was  the  development  of  minimal  software  for  connection 
routing,  setting  of  neuron  parameters,  synaptic  gains  and  time  constants  and  control  of 
the  A/D  conversion. 

A  third  objective  was  the  functional  evaluation  of  this  system  through  performance  of 
small  computational  tasks. 

The  first  two  of  these  objectives  were  fully  met  by  the  end  of  October.  By  that  date  the 
machine  had  been  assembled,  connections  and  synapse  parameters  could  be  set  by 
software,  the  output  multiplexer  was  functional  but  the  display  software  had  not  yet  been 
completed.  Consequently  only  a  few  very  simple  tests  involving  the  routing  of 
connections  between  neurons  and  the  summing  of  inputs  had  been  demonstrated.  By 
now  the  display  software  is  functional  and  a  few  further  examples  of  computational  tasks 
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Because  of  budget  constraints  the  size  of  this  prototype  had  to  be  limited  to  9 
neuron  modules  (  72  neurons).  18  synapse  modules  with  2448  synapses  and  72  routing 
switch  modules  (20736  switches). 

It  was  anticipated  that  by  the  end  of  Phase  I  a  finalized  design  had  emerged  that 
would  serve  as  the  basis  for  the  work  performed  during  Phase  II  aimed  at  the 
construction  of  a  full  scale  machine  and  the  development  of  extensive  control  and 
application  software  with  the  goal  to  create  a  commercial  product.  This  expectation  has 
been  met.  The  VLSI  components  and  modules  performed  according  to  specs  and  the 
Phase  II  machine  will  be  designed  essentially  as  a  scaled  up  version  of  the  Phase  I 
computer.  Photographs  of  the  prototype  are  shown  in  Figs.  1  and  2. 
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Fig. 2.  Photograph  of  the  prototype,  showing  the  main  circuit  boards,  I/O  buffers  , 
power  supplies  and  LED  display  panel  in  their  chassis.  Cables  are  ommitted. 

Details  of  progress 

The  Phase  I  contract  called  for  the  construction  of  a  small  Prototype  Neural  Analog 
computer  to  be  completed  by  10/31/90.  The  work  involved  several  separate  aspects 
that  were  being  pursued  concurrently.  They  included: 

A.  Fabrication  of  the  VLSI  modules  for  Neurons,  Synapses,  Time  constants  and 
Routing  Switches. 

B.  Design  and  fabrication  of  circuit  boards  for  the  mounting  and  interconnection  of 
the  modules. 

C.  Development  of  software  for  the  Network  Monitor. 

D.  Design  and  construction  of  a  digital  Network  Controller  for  setting  connection 
architectures,  neuron  parameters, synapse  weights  and  time  constants. 

E.  Development  of  software  for  low  level  Network  control. 

F.  Performance  of  simple  computational  tasks 

Progress  in  these  areas  through  11/31/1990  is  described  below. 
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A.  Fabrication  of  the  VLSI  Modules. 

Final  versions  of  test  chips  for  Neurons  ,  Switches  and  Synapses  were  submitted  for 
fabrication  in  May  and  returned  from  the  foundry  in  July. 

Following  the  testing  an  improved  scheme  for  the  synapse  was  designed,  fabricated 
and  tested.  The  redesign  was  necessary  because  the  digital  nature  of  the  synaptic  gain 
control  does  not  allow  straightforward  implementation  of  a  logarithmic  gain  scale.  If  each 
gain  is  controlled  by  one  bit  ,  the  total  gain  is  the  sum  of  5  individual  gains  each 
controlled  by  one  bit.  This  leads  inevitably  to  jumps  in  the  gain  curve  and  large  regions  of 
control  code  over  which  there  is  little  gain  change.  In  the  improved  design  a  linear  3  bit 
gain  is  multiplied  by  four  different  constants  controlled  by  the  4th  and  5th  bit.  This 
floating  point"  scheme  provides  a  linear  approximation  to  a  logarithmic  scale  over  4 
orders  of  magnitude  of  synaptic  gain  (see  Fig.5).  Test  of  linearity,  resolution  and 
accuracy  of  the  weights  are  shown  in  Table  1. 

A  test  chip  for  modifiable  time  constants  was  submitted  to  a  foundry  in  Europe 
and  was  successfully  tested,  a  full  scale  chip  was  submitted  to  MOSIS  at  the  beginning 
of  August  and  returned  at  the  beginning  of  October.  Since  we  were  not  certain  that  the 
VLSI  time  constant  circuit  would  work,  we  implemented  also  a  number  of  modifiable  time 
constant  circuits  using  macroscopic  ICs  as  backups.  These  circuits  are  plug  compatible 
with  the  VLSI  Chips  and  could  take  their  place  if  the  chips  had  not  worked.  However,  the 
VLSI  time  constants  did  work  and  are  integrated  into  the  network.  Representative 
outputs  from  the  circuit  for  square  wave  inputs  are  shown  in  Figs.8  and  9.  The  range  and 
value  of  the  time  constants  turned  out  to  be  larger  than  expected  from  simulations.  This 
would  allow  a  reduction  of  the  chip  size  for  the  Phase  II  version.  There  is  also  a  small 
systematic  DC  offset  for  the  largest  time  constants  which  can  be  compensated  by  an 
appropriate  neuron  bias  and  would  be  corrected  for  the  Phase  II  design. 

Extensive  performance  evaluation  of  the  test  chips  showed  that  switches,  synapses 
and  neuron  output  multiplexer  performed  as  specified  and  in  accordance  with  spice 
simulations  of  the  designs.  There  was  a  layout  error  in  the  neuron  design  which  had  to 
be  corrected. 

Following  the  tests,  full  scale  dies  for  the  production  runs  were  designed  and 
submitted  in  late  July  and  early  August.  All  chips  returned  from  the  foundry  during  the 
first  week  of  October  which  left  one  month  for  testing  and  final  assembly. 

After  submission  of  the  full  scale  synapse  chip  a  layout  error  was  detected  that 
renders  half  of  the  synapse  population  on  a  chip  nonfunctional.  The  error  was  corrected 
and  a  new  batch  was  submitted  which  returned  in  early  October. 

The  full  scale  synapse  and  switch  chips  are  too  large  for  manual  testing  and  special 
software-controlled  test  circuits  were  built  that  allow  automatic  testing  of  both  analog  and 
digital  portions  of  the  chips. 
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Overall  the  chips  performed  as  expected,  although  there  were  some  minor 
deviations  from  the  simulation  results. 

We  should  point  out  that  the  rate  of  our  progress,  especially  in  the  area  of  chip 
fabrication  was  limited  by  the  budget  as  well  as  the  fabrication  schedule  of  MOSIS  and 
the  turn-around  time  at  the  foundries  which  is  currently  in  the  order  of  2  months. 
Complete  simulation  of  the  full  scale  chips  is  not  possible  within  the  available  time  and  a 
design  fault  may  not  be  detected  until  after  fabrication  and  testing  in  which  case  another 
2  to  3  months  are  lost. 

Layouts,  photographs  and  some  test  results  for  the  full  scale  chips  are  shown  in 
Figs.  3  to  9  and  Table  1 . 
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Fig.  5.  Test  results  from  the  synapse  chip.  The  plot  shows  the  synaptic  gain  as  a 
function  of  the  control  code.  The  values  are  slightly  higher  than  expected  from 
simulations. 


Code 

Weight 

Mean 

Variance 

0 

+10 

12.9667 

1.6318E-2 

n 

+5 

6.6727 

5.1128E-3 

2 

+2.5 

3.2465 

3.7290E-4 

3 

+  1 

1.2414 

7 .2805E-5 

4 

+0.5 

6.1867E-1 

1.5646E-5 

5 

+0.25 

3.1602E-1 

4 .7350E-6 

6 

+0.1 

1.2630E-1 

7 .0322E-7 

7 

+0.05 

6.3817E-2 

1 .8371E-7 

8 

+0.025 

3.2549E-2 

8.4386E-8 

9 

+0.01 

1.3996E-2 

4 .8758E-8 

10 

+0.005 

7.4948E-3 

1.9904E-8 

11 

+0.0025 

3.9400E-3 

2 .1669E-8 

13 

-o.co:5 

-4.0227E-3 

9.2433E-8 

14 

-0.005 

-7.7870E-3 

3.1502E-7 

15 

-0.01 

-1.4620E-2 

7.3654E-7 

16 

-0.025 

-3.3265E-2 

3.0712E-6 

17 

-0.05 

-6.5201E-2 

9.1539E-6 

18 

-0.1 

-1.2911E-1 

2 .5840E-5 

19 

-0.25 

-3.2310E-1 

8.7545E-5 

20 

-0.5 

-6.3347E-1 

1.8959E-4 

21 

-1 

-1.2698 

4 .4117E-4 

22 

-2.5 

-3.3494 

1.0425E-3 

23 

-5 

-6.9482 

5.9766E-3 

24 

-10 

-13.7173 

1.7197E-2 

Table.  1.  Summary  of  measured  gains  and  their  statistics.  Only  24  of  64  possible 
values  are  shown. 


Fig.6.  Photograph  of  the  switch  module  chip. 
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Fig.  7.  Layout  of  the  time  constant  chip. 


Log  Time  Constant 


Fig.  8.  Responses  of  the  time  constant  circuit  to  square  waves  of  different  amplitudes 
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Fig.  9  .  Measurements  of  the  time  constants  as  a  function  of  the  control  code 
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B.  Design  of  the  Circuit  Boards. 

The  circuit  board  layout  was  determined  by  the  Chip  packages.  The  synapse  and 
switch  modules  were  packaged  in  84  pin  PGA’s.  The  neuron  modules  and  modifiable 
time  constants  are  packaged  in  40  pin  DIPs.  This  mix  of  packages  was  dictated  by  cost 
considerations  and  unfortunately  led  to  more  complex  interconnections.  The  circuit 
boards  were  designed  and  wirewrapped  during  the  second  half  of  September. 
Wirewrapping  was  chosen  instead  of  printed  circuits  because  of  cost  considerations  and 
because  wirewrapping  allows  for  easier  error  correction.  The  boards  were  tested  and 
minor  wrapping  errors  were  corrected. 

The  general  architecture  and  the  connection  schematic  of  the  boards  are  shown  in 
Figs.  10  and  11. 


Fig.  10.  Block  diagram  of  one  of  three  processor  boards  that  comprise  the  prototype. 
The  major  components  and  1/4  of  the  analog  data  paths  are  shown.  Support  chips, 
digital  data  buses,  and  power  connections  have  been  omitted.  The  boards  are 


connected  with  three  50  pm  connectors.  50  pin  connectors.  I/O  Buffers  and  digital 
interface  are  on  separate  boards 


Fig.  11.  Schematic  of  a  portion  of  the  connections  between  modules. 

C.  Network  Monitor. 

Selected  time  segments  of  the  neuron  outputs  are  multiplexed  on  the  neuron  chip, 
A/D  converted  and  subsequently  recorded,  stored  and  displayed  by  the  digital  host 
computer.  This  operation  which  proceeds  in  parallel  with  the  analog  processing  requires 
software  control  of  the  multiplexer  on  the  neuron  chip  and  an  A/D  board  in  the  host 
computer,  as  well  as  a  separate  graphics  program  for  the  display.  Software  available 
from  an  earlier  project  has  been  adapted  for  this  purpose.  In  addition  software  was 
developed  for  the  continuous  gray  scale  display  of  the  activity  of  all  neurons  on  the 
host  computer  screen.  Although  the  Phase  I  prototype  contains  an  LED  panel  for  the 
display  of  neuron  activity,  the  software  would  eliminate  the  need  for  a  hardware  display 
panel  in  a  larger  Phase  II  machine. 


D.  Development  of  the  Network  Controller. 

A  digital  interface  card  for  the  digital  loading  of  switch  settings  as  well  as  neuron  and 
synapse  parameters  was  built  and  tested.  This  card  configures  the  complete  state  of  the 
digital  portion  of  the  net.  i.e.  it  loads  the  switch  settings  and  synapse  parameters  acting 
as  interface  between  the  digital  host  computer  and  the  network.  It  also  reads  and 
verifies  the  actual  current  parameter  settings  in  the  neural  computer.  The  data  rate  for 
loading  the  parameters  is  currently  2  MHz.  Additionally  the  board  controls  the 
multiplexing  of  the  analog  outputs  from  any  or  all  neurons  for  the  network  monitor.  The 
design  is  such  that  most  of  the  operation  is  done  by  hardware,  so  that  there  is  little 
software  overhead.  The  network  controller  card  resides  in  the  host.  The  data  flow 
through  switch  and  synapse  module  and  the  schematic  of  the  interface  are  shown  in 
Figs.  12  and  13. 
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Fig.  1 2.  Data  flow  through  the  switch  and  synapse  module  for  loading  switch  settings 
and  synaptic  gains. 
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Fig.  13.  Schematic  of  the  digital  interface  card.  This  card  loads  the  connections  and 
parameter  settings  into  the  network.  It  is  housed  in  the  digital  host. 


An  overview  of  the  software  controlling  the  neural  computer  is  shown  in  Fig.  14. 
The  network  is  programmed  by  setting  the  routing  switches,  neuron  parameters, 
synapse  weights  and  time  constants.  Programming  can  be  achieved  at  several  levels, 
and  currently  only  the  lower  level  tools  have  been  completed.  At  the  lowest  level  the 
operator  sets  network  parameters  directly  from  the  keyboard  of  the  host  computer  using 
the  manual  router  programs.  A  graphic  display  of  the  network,  shown  in  Fig.  15, 
highlights  the  connections  and  displays  numerically  the  synapse  parameters.  The 
program  generates  arrays  of  the  parameters  that  are  then  loaded  into  the  network  by  the 
network  controller.  This  system  is  sufficient  for  the  Phase  I  prototype  and  two  version  of 
of  this  manual  programmer  have  been  implemented. 

For  the  Phase  II  version  a  high  level  language  is  under  development  that  allows  the 
design  and  graphic  display  of  a  conceptual  network  and  its  automatic  transposition  into 
the  neural  computer.  The  graphic  design  part  of  this  program  allows  the  design  of  the 
entire  network  or  segments  of  the  network  at  any  scale  and  generates  a  netlist  of 
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connections,  neuron  parameters  as  well  as  synaptic  weights  and  time  constants  which 
shall  eventtally  serve  as  a  database  for  programming  the  neural  net.  This  part  of  the 
pogram,  referred  to  as  Network  Design  in  Fig.  14,  has  been  written  and  was  used  to 
show  the  network  configurations  for  the  performance  examples.  However  it  has  not  yet 
been  interfaced  directly  with  the  neural  computer. 

We  should  point  out  that  a  complete  software  package  for  network  design, 
simulation  and  automatic  implementation  in  the  neural  computer  together  with  the 
implementation  of  various  learning  algorithms  lies  outside  the  scope  of  Phase  I  and 
would  become  available  only  during  Phase  II. 


17 


.  i , . . . 


■innnnHHHDHn^ 


.  {  .H  .  ^ 


■tiai  ■■■•!  ■■•■I 

■  nil  iiili  ■■ill  iiiii  iiiii 

••■■I  ■■■■•  ■•■■I 

•■■■I  ■■■■■  ■■t«l  ■«•■« 

•••• 

1.  : .  •  , 


■•  ■■■■I 
■I  •••«! 


•  if  ( " 

Hill  IIIII  IIIII  mil  mil  inn  iini 

■  mi  ■••■I  ■•■■I  •••ii  ■•••«  ■ii«i  ■•■•I 

■■■■I  imi  ■■■■•  ••■■I  ■•■■«  «■■■!  mil 

•••• 

» •  1. '  '  •  S.  • 


Rg.  15.  Display  screen  of  the  network  configuration  software  ( Manual  Router  in  Rg. 
14).  The  program  can  be  configured  for  networks  with  arbitrary  numbers  and  ratios  of 
component  modules.  It  is  menu  driven  and  displays  the  network  at  different  selected 
scales.  The  settings  of  connections,  sv/itches,  synapse  weights  and  time  constants  are 
set  by  a  mouse.  The  parameter  settings  are  stored  and  loaded  into  the  modules  through 
the  digital  interface. 


1?- .Display  screen  or  me  Network  Design  software.  The  program  is  menu 
anSaSh Iro^aiSf^  ®  network  at  different  arbitrary  scales.  Tne  connection 

individually  selected  and  stored  as  a  file  for 
mapping  and  loading  into  the  neural  computer. 


of  the  Network  Design  software.  The  progi 
all  of  a  network  at  different  arbitrary  scales.  In 
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Software  platforms 

The  different  software  routines  were  written  in  “C"  and  reside  in  the  digital  host.  They 
run  currently  under  DOS  or  Microsoft  Windows  on  386  machines,  however,  the  entire 
package  will  be  made  transportable  to  Unix  based  machines  so  that  more  powerful 
workstations  can  serve  as  hosts  for  the  larger  machines. 

The  modular  nature  of  the  neural  net  hardware  enables  us  to  write  the  software  routines 
in  a  format  that  allows  easy  expansion  or  adaptation  to  networks  of  different  and 
arbitrary  architecture  or  size. 


Performance  of  simple  Computational  tasks. 

We  have  programmed  the  machine  for  several  simple  tasks  that  allowed  us  to  evalute 
the  performance  of  the  routing  switches,  synaptic  weigths,  synaptic  time  constant  and 
neuron  parameters. 

A  few  examples  are  illustrated  m  Figs. 

The  figures  show  the  conceptual  networks,  for  some  cases  the  routing  configuration  in 
the  neural  computer  and  selected  performance  records. 

The  programming  software  used  for  setting  the  network  configurations  was  a  lower  level 
version  of  that  shown  in  fig.  . 

The  first  network  shown  in  Fig.  17  illustrates  a  fan  -  out  ,  fan  -  in  net  and  demonstrates 
that  small  signals  generated  by  small  synaptic  weights  can  be  transmitted  and  summed 
without  distortion  of  the  original  waveform  by  the  small-signal  noise. 

The  second  example,  (Fig.  18).  is  a  "Winner  Take  AH"  net  which,  depending  on  the 
inhibitory  feedback  gain,  either  extracts  the  largest  output  among  16  neurons  or 
enhances  the  contrast  between  the  different  outputs.  The  network  settles  within  the  time 
constant  of  the  neuron  outputs  (here  SCX)  us)  and  showed  no  oscillations.  Simulations  of 
this  small  network  on  a  SUN  4/110  are  slower  by  a  factor  of  100.  Since  this  is  a  fully 
connected  net  the  speed  ratio  would  scale  with  N2  of  the  number  of  neurons. 


Fig.  17.  Description  on  next  page. 
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Fig.  17.  A  network  demonstrating  the  fan  -  in  /  fan  -out  capabilities  ot  the  machine.  The 
conceptual  net  is  shown  at  left.  A  single  external  input  is  distributed  with  a  synaptic  gain 
of  0.03  from  an  input  neuron  to  32  secondary  neurons  (only  28  are  drawn)  and  their 
outputs  are  summed  at  a  single  third  stage  neuron  with  a  gain  of  1 .  The  records  show 
from  top  to  bottom;  the  input  signal,  the  output  of  the  first  neuron,  an  output  from  a 
second  stage  neuron  and  the  output  from  the  final  stage  neuron. 

Notice  that  the  low  signal  noise  from  the  second  stage  neurons  averages  out  at  the  third 
stage. 


The  third  network  shown  in  Rg.  19  is  a  set  of  8  coupled  neuron  oscillators  that  involves 
inhibitory  weights  with  longer  time  constants  ^an  the  excitatory  weights.  It  demonstrates 
the  proper  operation  of  the  time  constant  circuits  and  shows  that  dynamic  patterns  can 
be  generated  entirely  from  within  the  net.  In  the  simplest  case  a  single  neuron  can  be 
configured  as  a  relaxation  oscillator  and  made  to  generate  rhytmic  activity  analogous  to 
action  potentials  in  biological  neurons  solely  by  feedback  excitation  and  inhibition  with 
appropriate  time  constants.  The  waveform  and  oscillation  frequency  of  this  activity  are 
controlled  by  the  weights,  time  constants  and  threshold  bias  of  the  circuit. 

Several  other  small  circuits,  not  illustrated  here,  have  been  programmed,  including  an 
associative  network  of  24  neurons,  a  motor  control  circuit  that  models  the  last  stage  of 
the  control  of  saccadic  eye  movements,  a  neural  integrator  and  several  circuits  for  the 
computation  of  time  domain  pattern  primitives. 

A  more  elaborate  network,  involving  most  of  the  available  neurons,  performs  an  initial 
decomposition  of  acoustical  patterns.  It  receives  input  from  a  set  of  8  bandpass  filters 
and  extracts  local  maxima  of  amplitude  vs  frequency,  (d2E/dS2),  local  rates  of  rise  and 
fall  of  amplitude  (  =,-  dE/Dt),  and  local  rise  and  fall  of  frequency  i.e.  motion  (dS/dT)  of 
activity  along  the  frequency  axis.  This  network  is  illustrated  in  Fig.20.  Other  networks  are 
currently  being  designed  and  tested. 


Fig.  18  A.  Conceptual  network  ( top)  and  the  routing  configuration  (bottom)  of  a  Winner 
Take  All  Net.  A  single  external  input  is  distributed  with  different  positive  gains  to  16 
secondary  neurons  whicn  are  connected  by  mutual  inhibitory  connections. 


Fig.  18  B.  Performance  of  the  Wnner  Take  AM  network.  Outputs  from  16  secondary 
neurons  in  response  to  a  square  wave  input.  In  A  the  inhibitory  gains  were  set  to  zero.  In 
B  these  gains  were  0.5  and  in  C  they  had  avalue  of  0.9.  Notice  that  the  0.5  inhibition 
results  in  a  contrast  enhancement.  The  records  were  obtained  through  the  analog 
multiplexers  on  the  neuron  modules. 
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Rg.  20.  Section  of  the  conceptual  design  of  a  network  for  the  primary  decomposition  of 
acoustical  pattems.The  primary  neurons  receive  input  from  eight  band  pass  filters. 
These  neurons  are  connected  with  mutually  inhibitory  inputs  in  a  center-surround 
scheme  with  spatially  decaying  gains.  They  extract  the  maxima  of  the  sound  amplitude. 
The  next  stage  extracts  separately  the  temporal  rise  and  decay  of  the  sound  amplitudes. 
These  neurons  receive  delayed  excitatory  or  inhibitory  inputs  from  the  previous  stage. 
The  third  stage  neurons  are  normally  "on“  through  positive  bias  input  and  compute  the 
complement  of  the  activity  of  the  second  stage  neurons.  The  fourth  stage  units  compute 
the  changes  of  frequency  maxima  and  their  direction  through  a  combination  of  the 
second  and  third  stage  neurons.  In  essence  they  are  motion  detectors.  In  the  physical 
network  ail  computations  are  performed  in  parallel  and  in  real  time. 
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Fig.20.B.  Neuron  outputs  from  the  network  shown  in  Fig.  20  A  for  four  different 
phonemes  illustrating  the  differences  of  the  patterns.  A  detailed  discussion  of  this  data 
is  beyond  the  scope  of  this  report. 
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Several  conclusions  can  be  drawn  from  these  tests.  First  of  all  the  machine  performance 
has  exceeded  ail  expectations  regarding  noise  levels,  accuracy,  stability  and 
programming  flexibility.  Particularly  the  routing  space  proved  entirely  adequate.  There  is 
therefore  no  reason  to  project  an  increase  in  the  ratio  of  switch  modules  to  neuron  and 
synapse  modules.  No  major  design  change  for  the  neurons  is  needed.  The  synapse 
gain  (weighting)  scheme  should  be  improved  by  increasing  the  resolution  for  middle 
range  gains  (between  0.1  and  1.  But  this  can  be  done  within  the  current  6  bit  resolution. 
The  only  limitation  for  temporal  pattern  computations  is  currently  the  low  ratio  of 
programmable  time  constants  to  synapses  and  this  problem  shall  be  addressed  with 
high  priority  during  Phase  II. 


